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ABSTRACT 



This paper reports on the formative evaluation results of 
the project "Enhancement of Problem Solving and Scientific Reasoning Skills 
through Computerized General Chemistry Modules." This project includes the 
design, development, and implementation of two interactive computer modules 
for the enhancement of mathematical and problem solving skills of 
undergraduate students who are taking general chemistry courses at the 
University of Puerto Rico. Studies have shown that there is a correlation 
between students' performance and their math skills. The modules, 
Introductory and Equilibrium, were designed and written in Spanish, then 
translated into programming language. The Introductory module includes math 
skills, graphs, and problem solving. For the evaluation process, the first 
trial used a single experimental group with a pre-post test while the second 
trial used a pre- and post- test between an experimental and control group. 
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Summary 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



This report details the formative evaluation results of the pilot implementation of the 
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project Enhancement of problem solving and scientific reasoning skills through computerized 



General Chemistry modules. This project consists in designing, developing and implementing 



two interactive computer modules. Introductory and Chemical Equilibrium, for enhancing 



mathematical and problem solving skills in undergraduate students taking the General Chemistry 



course at the University of Puerto Rico, Rio Piedras Campus (UPR-RP). Both modules were 



designed and written in Spanish by the two faculty coordinators of the project (the second and 
third author of this paper) and translated to a programming language and software format by the 
company ACO Digital. Both modules are written in Spanish and are available upon request. 



The evaluation process was undertaken during the pilot implementation of the 



Introductory module in two trials or stages during the academic year 1999-2000. In the first trial 



(first semester), a sample of 10 voluntary undergraduate students enrolled in a section of General 



Chemistry at the UPR-RP completed the work on the module in 17 computer sessions. In the 
second trial (second semester), a group of 32 General Chemistry’s students used the module in 
54 sessions. In both sections, the students worked individually through the module in the 
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1 This proyect was funded by a grant from the Fund for the Improvement of Post Secondary Education (FIPSE) of the 
Department of Education of the United States and the University of Puerto Rico, President’s Office. 

Maria del R. Medina-Diaz was the external evaluator, and Francisco Echegaray and Noel Motta, are the project’s 
coordinators. 
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scheduled time. In all sessions, one of the project’s coordinator was present and assisted the 
students. 

The main results of this evaluation were the following: 

1 . The students spent approximately a total of six hours working on the Introductory 
module. 

2. In both trials, all users indicated that the module’s directions and explanations were 
clear, the examples were helpful and the vocabulary was easy to understand. 

3. They used all or almost all of the information and helping devices in the screens. 

4. Eighty-percent of the students praised the module as excellent because of its features. 
However, one of the features that most disliked was the text extension of the introductory 
explanations. 

5. The students were interested in their work on the module and they highlighted that it 
helped to improve their math skills. 

6. The students made several suggestions to improve the module and these were solved. 

7. Non-significant differences were found in the pre- and post-test scores of the single 
sample. In the second trial, significant differences were found between the post-mean 
scores of the experimental and comparison group in the mathematics test. 

8. Non-significant differences were found between the experimental and comparison group 
in the final exam but the mean score of the experimental group was higher than of the 
comparison group. 

However, it should be noticed that some variables may influence the test scores of both 

groups: (1) different teaching styles, (2) sample selection and assignment to the groups, 

(3) changes in testing, (4) students’ attitudes towards treatment, and (5) treatment briefly 
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applied. Some recommendations are also included. 

Introduction 

Evaluation of instructional technology is essentially no different from evaluation of any 
sort of teaching and learning, and it should not be a casual process. In the field of educational 
evaluation, the evaluation of the student learning has been the most salient endeavor. There is 
often little attention and evaluation to the teachers' course materials and their usage in the 
classroom. As instructional technology has evolved, it has been subject to evaluation scrutiny. 
Today, more classroom teachers are using different kinds of technologies as instructional tools 
and it is necessary to evaluate their effectiveness in a systematic way. 

According to Hancock (1977), evaluation of instructional technology must describe, as 
much is possible, what is happening; suggest possible changes that may lead to its improvement; 
and provide adequate information to assist the policy-making. Such evaluation can be 
performed with two approaches: formative and summative. Formative evaluation provides 
information about how well the instructional program is implemented and this information can 
be used for reaching conclusions regarding its merits. This evaluation aids to determine its 
weaknesses so that revisions can be made to make it more effective. 

On the other hand, summative evaluation primarily focuses on the end product or 
outcome of the instruction. It provides an understanding of how a given instructional program 
reaches its goals and the findings assist the decision-makers to take decisions regarding its 
continuation. Steinberg (1991) points out that documentation regarding summative evaluation 
rarely accompanies computer and other technology lessons. The same tradition sustains for the 
minimal evaluation of other instructional materials such as textbooks. 




4 



4 



In this instance, a formative evaluation was conducted during the pilot implementation of 
the Introductory module, which is part of project Enhancement of problem solving and scientific 
reasoning skills through computerized General Chemistry module. The focus of this evaluation 
was on determining the extent to which the project activities were conducted as planned and on 
providing information of its progress towards the goals and objectives. The evaluation was 
conducted for an external evaluator with strong collaboration of the project coordinators. In 
order to pursue the formative evaluation process, the following steps were taken: (1) Establishing 
the project’s goals, (2) Formulating the evaluation questions, (3) Selecting the design of 
evaluation, (4) Developing data collection instruments, (5) Collecting and analyzing data, and 
(6) Reporting results, conclusions and recommendations. These steps are similar to those 
suggested by Fitz-Gibbon & Morris (1987); King, Moris & Fitz-Gibbon(1987); Popham (1988); 
and Worthen, Sanders & Fitzpatrick ( 1997). 

Other approaches have been proposed to pursue formative evaluation of instructional 
products. For example. Smith & Ragan (1993) have elaborated a comprehensive four-stage 

procedure which includes (1) design review (revisions of goals, learner and context analysis, task 

/ 

analysis and specifications prior the development of the material in order to confirm the 
accuracy of the design); (2) expert reviews (content revisions of various specialists in order to 
check the accuracy and completeness of the material); (3) learner validation (tryout the material 
with a small group of the target audience for determining and rectifying problems in the 
instruction delivery); and (4) ongoing evaluation (subsequent revisions of the material usage 
according to the changes in the instructional context). However if time and money are short, the 
formative evaluation may include at least learner validation or field-testing. Accordingly, the 
stages 1, and 2 were performed in the development of the computerized modules. The Smith & 
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Ragan’s stage 3, learner validation, is the crux of the formative evaluation of this project. 

Project description 

The project Enhancement of problem solving and scientific reasoning skills through 
computerized General Chemistry modules was an intitative of the two coordinators to help 
improve student performance in the General Chemistry Course. Studies in the UPR-RP 
Chemistry Department have shown that student performance in this course was significantly 
related to the students math skills. Also in other universities, problem-solving skills have been 
identified as one of the main barriers for reaching a satisfactory performance in the General 
Chemistry course. During the last two decades, the Chemistry Department of the UPR-RP has 
implemented various programs directed to increase students' success in the General Chemistry 
course such as the Personalized System of Instruction, and Pre-chemistry and Study Skills 
workshops. This project is the most recent and cost-effective initiative for reaching out students 
who need to improve math and problem solving skills. It includes designing, developing and 
implementing two interactive computer modules (Introductory, which includes Math Skills, 
Graphs and Problem Solving, and the Chemical Equilibrium ) with the intent of enhancing 
problem solving skills and critical thinking in a context of solving numerical and conceptual 
problems. 

The intention of the Introductory module is to provide students the necessary 
mathematical skills and guidelines for solving problems related to General Chemistry. It is 
divided in two tutorial sections written in Spanish. Math Skills and Graphs, and Problem 
solving. The Math Skills and Graphs section presents basic content information ranging from 2 
to 8 pages and includes from 2 to 6 problems related to each of the following topics: ratios and 
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proportions, percentages, algebraic equations, logarithms, scientific notation and graphs (See 
Table 1). All of the working screens with the problems also have available the following 
information devices: "Factores de conversion" (Conversion factors), "Formulas" (Formulae), 
"Tabla periodica" (Periodic table), and "Constantes fisicas y quimicas" (Physical and chemical 
constants). Helping device buttoms such as, “Sugerencias” (Suggestions) and "Dedo de la 
mano" (Hand finger), offer suggestions for solving the problems in this section. 



Table 1. Components of the Introductory module 



Introductory module 
sections 


Number of introductory 
pages per section 


Number of exercises 


1. Math skills 

- Ratio and proportions 


3 


4 


- Percentages 


2 


4 


- Algebraic equations 


2 


4 


- Logarithms 


4 


6 


- Scientific notation 


4 


4 


- Graphs 


8 


2 


2. Problem Solving 


12 


9 


Totals 


35 


33 



The purpose of the Math Skills part is that students review and apply the basic math 
algorithms related to scientific notation, ratio and proportions, percentages, algebraic equations, 
logarithms in a set of problems to be solved. The reviews included examples are provided to 
illustrate the concepts and procedures. The Graphs part of the module emphasizes data 
representation and interpretation. 

The Problem Solving section includes an introduction menu with the skills required to 
solve problems: (1) "Extraction de datos y visualization" (Data extraction and visualization). 
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(2) "Reconocimiento de incognita" (Identify the unknown), (3) "Analisis y desarrollo de la 
estrategia" (Analysis and Strategy Development), (4) "Planteamiento de la solution 
escrita"(Statement of a solution), and (5) "Resolution matematica y metacognicion" 
(Mathematical solution and metacognition). Each of these skills is explained in several pages 
and examples are provided, some of these being interactive. Also, this section contains nine 
practice problems. All the working screens, contain information and helping devices. The 
information options were the same as in the Math skills section. The help buttons were 
"Bombilla" (Light bulb) for extracting data, "Lapiz" (Pencil) for making a diagram, and "Nube" 
(Cloud) for analysis questions. 

Project’s Goals, objectives and activities 

The project's goals and objectives are the following: 

Goal #1 : Improve academic performance of the UPR-RP students in the General Chemistry 
course. 

Objective #1: Implement modules in the General Chemistry course 
Goal #2 : Improve academic performance of transfer students 
Objective #2: Provide access to other institutions. 

In order to achieve these, the following main activities were planned: 

1) Design and prepare modules in computer language 

2) Provide access to modules to students in a pilot group 

a) Set up computers with the modules 

b) Write a guide for the general use of the modules 

c) Training students in the use of modules 
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Evaluation questions 

The set of questions that guided the pilot study of the project were the following: 

1. Were the project activities implemented as intended and according to the proposed timeline? 
If not, what changes were needed in the project implementation? 

2. How much time did it take for students to complete the module? 

3. Did the students complete the module successfully within the scheduled time? 

4. What were the students' opinions about the module's features? 

5. How relevant was the use of the module for students taking the General Chemistry course? 

6. Overall, how did the students evaluate the module? 

7. Did the students improve their scores from the pre-test to the post-test? 

8. How was the students' performance in the final exam and in the General Chemistry course? 



Evaluation desing 

In order to answer these questions two designs were used for evaluating formatively the 
Introductory module implementation: (1) a single group pre-post test in the first trial, and (2) a 
pre- and post- test between an experimental and comparison group in the second trial. 

In the first trial, a group of 21 undergraduate students enrolled in a section of General 
Chemistry voluntered for using the module at least once. They worked individualy through the 
module weekly in the scheduled hours. Ten out of the 21 (48%) students successfully completed 
the Introductory module in the first semester. All of these were first or second year students in a 
Natural Sciences undergraduate program. In the second semester, 32 out of 38 (84%) students in 
a section of General Chemistry used the module ("Experimental group"). Another section of 
twenty-one General Chemistry students was the "Comparison group". Both groups included 
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second, third and four year students in the Natural Sciences and Education undergraduate 
programs. According to the following data, both groups were similar in their math and science 
background (See Table 2). The coordinators informed the students about the purpose of the 
project and they signed a consent form to participate. 



Table 2, Characteristics of the students in the second trial 



Characteristics 


Groups 






Comparison 


Experimental 


Undergraduate major: 


Natural Sciences 


10 (48%) 


21 (72%) 


Education 


11 (52%) 


8 (38%) 


Year of sudy: 


First 


3 (14%) 


10(31%) 


Second 


4 (19%) 


8 (25%) 


Third 


6 (29%) 


6(19%) 


Fourth 


3 (14%) 


4 (13%) 


Fifth or more 


5 (24%) 


4(13%) 


Courses approved: 


Pre-calculus 


20 (95%) 


20 (69%) 


Calculus 


6 (29%) 


13 (45%) 


Other Science courses 


19(90 %) 


4(13%) 


Times in the General Chemistry 


course: 




First time 


6 (29%) 


8 (28%) 


Second time 


12 (57%) 


16(55%) 


Third time 


2 ( 9%) 


5(17%) 


Fourth time 


1 (29%) 


0 (0%) 


Total sample 


21 


29* 


Note. * Twentv nine out of 32 answered the questions. 





Data collection instruments 

A brief description of all the instruments used for the formative evaluation follows. A 
mixed-method approach was employed combining quantitative and qualitative data collection 
instruments. All of the instruments were written and answered in Spanish. They are available 
upon request. 
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1 . Attendance sheet 

Every time a student attended to the computer sessions, the professor in charge checked 
the student name in the Attendance Sheet and the part of the module that he or she completed. 
This sheet helps to keep track of the sections and problems where each student was working. 

2. Instructor's observations notes 

In order to find out areas of weaknesses for improving the module execution, the students 
were observed and asked to point out to the professor in charge any problem, error or 
misunderstanding while they were working on the module. The professor in charge took notes 
of the student’s findings and of his owns. The two professors in charge of attending to the 
sessions completed an observation data sheet. 

3. Pre- and post-test 

In the first trial, a pre- and post-test of math skills was administered to the students. It 
included a total of 30 multiple-choice items divided in six sections. Five items represented each 
of the six topics: Scientific notation. Equations, Ratio and proportions. Percentages, Logarithms 
and Graphics. The total score of both tests was 30. The order of some items and of the two 
parts of the pre-test was changed in the post-test. However, both tests covered the same topics 
and skills, and was based on the same set of specifications (See Table 3 ). 
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Table 3, Pre- and post-test blueprint in the first and second trial 



Test topics 


Skills 


Number of 
items 

(first trial) 


Number of 
items (second 
trial) 


1. Scientific 
notation 


-Express a number in scientific notation 
and vice versa 


5 


4 


2. Equations 


-Find solutions to linear and quadratic 
equations 


5 


4 


3. Ratios and 
Proportions 


-Find one value in the proportion 


5 


4 


4. Percentages 


-Determine the percentage of an amount 
-Determine the number related to a 
percentage 

- Find the total amount 


5 


4 


5. Logarithms 


-Find the logarithm and antilog of an 
expression 

-Identify logarithms rules 
- Solving equations with logarithms 


5 


4 


6. Graphs 


-Interpret information 
- Identify the appropriate scale 
-Identify variables and coordinates 


5 


5 


7. Problem 
solving 


- Data extraction 

- Identify unknown 

- Develop a strategy 

- Solving the problem 

- Revising the solution 


0 


5 


Total 




30 


30 



Because of the project’s objective in determining the effect of module use on students’ 
problem solving skills, the pre- and post-test were modified to include five problem-solving 
items in the second trial. The number of items in the five math skills was reduced to four, and 
five items represented graphs and problem solving skills. Alternate forms were administered as 
pre- and post-test to both groups. These forms have similar mean and standard deviation. 
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4. Interview 

In the first trial, two samples of ten students were interviewed individually after they 
finished working on the modules. The first sample of ten students was selected at random after 
they completed the Math section of the Introductory module. The other group of interviewers 
was all the students who completed the Problem Solving section. Twenty-two questions were 
included in the interview questionnaire for collecting students' data about module's features and 
operation. The interview setting was the backside of the computer room and lasted 
approximately 15 minutes. 

A sample of 16 out of 32 students was interviewed in the second trial. The interview was 
performed individually after they completed the work on the Problem solving section of the 
module, and scheduled at the student convenience. The interview questions were the same as 
before and the interviews were held in the computer room. 

5. Users' evaluation questionnaire 

A 29-item questionnaire was designed to collect information with regard to users' opinion 
about the Introductory module quality and usefulness. In the first trial, this instrument was 
administered at the end of the semester and answered by 1 1 students who used both sections of 
the module. During the second semester, 28 out of 32 students (88%), who completed the 
module, answered the questionnaire. 
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Data analysis 

The responses of the two samples of students, who completed the work through the 
Introductory module in the first and second trial, were used in the statistical and the narrative 
data analysis. In the second trial, the pre- and post-test data of the 32 students were compared 
with a cohort group of 21 students. Descriptive statistics, Cronbach’s internal consistency 
reliability indexes and item difficulties were calculated for each test. Also, Pearson’s correlation 
indexes were computed using the students’ scores in the pre- and post-test, final exam and 
course grade. A t-test was performed for determining the differences in mean scores of the 
experimental and comparison groups. The program GB-STAT for Mac was used in the statistical 
analysis. 



Results 

This section of the report includes the relevant results and interpretations related to the 
evaluation questions proposed. These are identified by the relevant variables of the questions. 

Project activities and changes made 

The main activities for the first year of the project were performed as planned: (1) to 
desing and develop the Introductory module, and (2) to provide access to students. The 
Introductory module (which includes Math Skills, Graphs and Problem Solving ) was designed 
by the coordinators of the project on time, and translated to a computer-programming language 
accordingly by the company ACO Digital. The Introductory module is available in PC format 
upon request and the Chemical Equilibrium module is in trial period. 

During the academic year 1999-2000, the Introductory module was pilot-tested with two 
groups of General Chemistry students at UPR-RP. In the first semester, 21 undergraduate 
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students enrolled in a section of General Chemistry voluntarily accepted to participate in the 
project. Of these only 13 worked partially or completely on the Problem solving section, the last 
part of the Introductory module. A total of 17 working sessions were held in this trial, and lasted 
two months (from October 4 through December 9 of 1999). These sessions were scheduled for 
two-hour periods on three-week days using PCs at the Natural Sciences Computer Center. 

A total of 54 sessions were held during the second semester (from January 3 1 to March 
24, 2000). Thirty-two out of 38 students (84%) completed the work on the module. In the 
second trial, the students used the I-MAC computers available in the Chemistry Department's 
Learning Resources Room. This room was specially prepared for the project and had 14 
computers with the Introductory module installed. Twelve hours per week were scheduled for 
working on the module. 

In both trials, every student sat individually at a desktop computer with the Introductory 
module installed in the computers CPUs. At the computer desk, each student was instructed to 
start the module and worked through individually. In the first trial, there were not written 
directions or related handouts. For the second trial, the professors wrote an instructions sheet to 
place besides each computer. The students were instructed to ask for the assistance of the 
professor in charge every time she or he needed and to point out any problem, error or 
misunderstanding that they found. In each trial, one of the coordinators was in the computer 
room. If the students had a question or difficulty raised a hand or asked the professor for 
assistance. Students used either their own calculator or the one provided in the module, a piece 
of paper (or notebook) and a pencil for doing the calculations. 

Every time a student attended to a session, the professor in charge (Dr. Echegaray or Dr. 
Motta) checked the Attendance Sheet in order to verify which part of the module she or he was 
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working on. She or he continued working through the module in the part where it was left. 

When the student finished to use the module, he or she notified the professor which part or 
problem completed. In the event that the student completed the Introductory module, she or he 
could leave the room or stay to answer the interview with the evaluator. 

Students' completion rate and time in the module 

In the first trial, the average of student attendance to the computer sessions was 4. The 
minimum number of students who attended to the sessions was 2 and the maximum 7. Eighteen 
students completed the Math Skills and Graph section. Ten students completed the work with 
the Problem Solving section by the end on the semester. However, three students did not finished 
but worked through at least five out of the nine problems. Students spent an average of 3 and 2.5 
hours working on the Math Skills and Problem Solving sections of the module, respectively. In 
the second trial, the mean student attendance was approximately 3 and ranged from 16 to 0 
students. By mid-semester, 32 finished the entire module. They spent approximately 3 and 3.5 
hours, respectively, working on each section of the module. 

Students' opinion about the module's features 

In both trials, all students who used the Introductory module indicated that the 
instructions and explanations were clear, the examples were helpful and the vocabulary was easy 
to understand. The students indicated that they used all or almost all of the available 
information and helping devices in the screen. The most used devices in the Math skills section 
were “Factores de conversion” (Conversion factors), “Sugerencias” (Suggestion), “Calculadora” 
(Calculator) and “Formulas matematicas” (Math formulae). In the Problem Solving section, they 
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used more frequently “Factores de conversion”(Convertion factors), “Analisis” (Analysis), 
“Datos” (Data) and “Diagrama” (Diagram). 

The intelligent feedback system was the most appraised feature of the module. About 
90% of the students in both trials commented that the feedback messages were helpful (“ayudo”) 
in giving some direction for reaching the correct answer. Table 4 shows a summary of the 
module’s features and the percentage of students’ responses in each trial. 

Most of the users found that the exercises and problems were of medium difficulty ("ni 
faciles ni dificiles"). In the Math Skills part, logarithms exercises were the most difficult for 
82% of the students. For 50% of the students, scientific notation was the easiest part. In the 
Problem Solving part, students' opinion regarding the most difficult problem varied from 
problems 1, 2 and 3. 

Relevance of the module for the students 

More than 90% of the students in the first trial indicated that they were either very 
interested or interested in their work on the Math skills and Problem Solving sections. Sixty and 
fifty percent of the students in the first and second trial, respectively, choose that they were very 
interested in the Problem Solving section. In the two trials, nine out of ten students indicated 
that both sections of the module helped them a lot ("mucho" and "bastante") in improving their 
math and problem solving skills. For instance, two students commented that the module helped 
them "to refresh and improve the skills learned before and in the class". Most of the students 
commented that the Problem Solving section was the most helpful for the General Chemistry 
course. Because the format of the problems were similar to those encountered in the course. 
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Table 4 . Student’s percentage of agreement or responses to the module features 
in each trial. 



Module features 


Math skills and 
Graph section 


Problem solving 
section 


1 . Instructions easy to understand 


100 1 


100 




100 2 


100 


2. Very clear or clear explanations 


100 


100 




79 


82 


3. Very useful or useful examples 


100 


100 




100 


89 


4. Fair amount of examples or problems 


90 


90 




96 


100 


5. Vocabulary easy to understand 


100 


70 




96 


85 


6. Usage of all or almost all helping 


100 


100 


devices 


100 


100 


7. Good response space 


100 


100 




100 


100 


8. Very helpful feedback 


91 


78 




89 


82 


9. Very attractive or attractive menu 


100 


100 




100 


96 


10. Organized screen format 


90 


100 




100 


100 


1 1 . Good screen sequence 


90 


100 




93 


93 


12. Fair or good amount of text in the 


80 


90 


screen 


77 


60 


13. Very helpful diagrams 


80 


90 




68 


79 


14. User is in control 


100 


100 




93 


93 


15. Fair extension (length) 


55 


40 




86 


50 


16. User very interested or interested 


91 


100 




89 


86 



1 Students responses in the first trial. 

2 Students responses in the second trial. 
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Students' evaluation of the module 

Eighty percent of the students praised the module as excellent. The main features that 
they liked the most were the explanations, aids, screen display and color, drawings and feedback. 
One of the features that most disliked was the text extension (length) in the sections. In the 
interview responses, the students' comments were positive with regard to the Math skills section 
of the module: -"Es bueno para repasar las destrezas matematicas" (It is good to review math 
skills), "Se lo recomendaria a otros estudiantes que no saben matematicas” ( I recommend it to 
students who don’t know math). The most salient comment of the students with regard to the 
problem solving section were the drawings and its capability of making drawings related to the 
problem. 

Students’ suggestions to improve the module 

Among the students suggestions to improve the Introductory module are the following: 

1) A short test at the end of each section or at the end of the module, 

2) More graphs problems and in other math skills, 

3) Directions for using the module's calculator, 

4) Directions for entering the responses, 

5) Problems in order of difficulty, 

6) Problems in other format, and 

7) Explanations related to choose the appropriate graph. 

These suggestions has been incorporated in the last version of the module. 
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Students' pre- and post-test scores 

First trial 

The pre-test was administered to 21 students and the post-test was answered by 13 
students (who finished at least five problems in the Problem solving section). Both tests were 
administered during an extra period of class and lasted approximately 80 minutes. Using this 
very small sample, the Cronbach’s internal consistency reliability indexes of the pre- and post- 
test scores were 0.78 and 0.84, respectively. The pre-test difficulty mean was approximately 
0.70. and the post-test difficulty mean was near 77%. Although the test was relatively easy for 
the students in both test administrations. Logarithms and Graphics were the most difficult topics. 
The item difficulty means for each of the test parts are shown in Table 5. Table 6 shows the 
correlation indexes, mean and standard deviation (SD) of the participants’ scores in the first trial. 



Table 5, Pre- and post-test item difficulty means in the first trial 



Test part 


Item 


Pre-test item 


Post-test item 




Identification 


difficulty mean 


difficulty mean 


Scientific notation 


1-5 


97 


92 


Equations 


6-10 


79 


85 


Proportions 


11-15 


88 


89 


Percentages 


16-20 


69 


88 


Logarithms 


21-25 


40 


66 


Graphs 


26-30 


45 


43 


Overall test difficulty 


mean 


69.7 


77.2 




20 



20 



Table 6 . Correlation indexes, mean and standard deviation (SD) of the scores in the first trial 



Scores 


Pre-test 


Post-test 


Final exam 


Course 

grade 


Mean 


SD 


Pre-test 


1.00 


0.18 


0.75 


0.86 


20.1 


3.41 


Post-test 


0.18 


1.00 


0.31 


0.46 


21.4 


3.86 


Final exam 


0.75 


0.31 


1.00 


0.84 


116 


15.53 


Course grade 


0.86 


0.46 


0.84 


1.00 


77.21 


9.13 



Non-significant differences were found between the pre- and post-test mean scores of the 
students in the six test-parts and overall. The scores on the pre-test correlate high and 
significantly with the final exam (r=0.75, p <0.05) and the course grade (r=0.86, p<0.01). Non- 
significant correlation indexes were found between the post-test data and the other scores. 
Second trial 

The pre- and post-test item responses of 32 students who worked through the 
Introductory module and of the 21 students in the comparison group were used in the statistical 
analysis. The descriptive statistics in the pre- and post-test of the experimental and comparison 
group were the following: 



Pre-test Post-test Final exam 



Group 


N 


Mean 


SD 


Mean 


SD 


Mean 


SD 


Experimental 


32 


16.06 


4.58 


19.06 


4.38 


80.42 


21.51 


Comparison 


21 


14.81 


4.93 


15.14 


4.75 


67.78 


24.40 




21 



21 



The pre-test was administered to both groups in the first week of class and the post-test 
by mid-semester. The Cronbach’s internal consistency reliability indexes were 0.76 in the pre- 
test and 0.70 in the post-test for the experimental group. For the comparison group the internal 
consistency reliability index of the scores was 0.80 in the pre- and 0.82 in the post- test. These 
statistics indicate a consistent pattern of students’ responses to the tests, and hence, reliable test 
scores. 

Table 7 shows the test-parts and their item difficulty means for the experimental and 
comparison group. The overall difficulty mean of the pre-test was 53% and 50% for the 
experimental and comparison group, respectively. This represents a medium difficulty pre-test 
for both groups. In the pre-test, the percentage of correct responses for both groups was closer in 
two out of seven test-parts (Equations and Problem solving). The most difficult parts in the pre- 
test for both groups were Logarithms and Problem solving. 

The post-test overall mean difficulties for the experimental and comparison group are 
quite distant (64% and 50%, respectively). Apparently, for the experimental group the post-test 
was a bit easier than for the comparison group. In all the post-test parts, the percentage of 
correct responses of the experimental group was higher than of the comparison group. In the 
experimental group there is an increment in the percentage of students who answered correctly 
the test items in five parts, except the Ratio and proportion, and Graph items. 
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Table 7 . Pre- and post-test results of the experimental and comparison group in the 
second trial 



Test part 


Item identification 


Pre-test item 
difficulty mean 


Post-test item 
difficulty mean 


Scientific notation 


1-4 


72 a 


89 






75 b 


75 


Equations 


5-8 


61 


72 






61 


65 


Ratio and proportion 


9-12 


80 


77 






75 


73 


Percentages 


13-16 


55 


78 






50 


50 


Logarithms 


17-20 


30 


50 






24 


35 


Graphs 


21-24 


48 


44 






45 


37 


Problem solving 


25-30 


27 


50 






28 


31 


Overall difficulty 




53 


64 


mean 




50 


50 



A Experimental group; B Comparison group 



The t-tests statistics for pairwise comparisons showed significant differences between the 
means of the experimental and comparison groups in the post-test scores (t=3.08, df= 51, p<.01). 
Also, significant differences were found in the means of the experimental group pre- and post- 
tests scores (t=-2.68, df=31, p<0.05). Non-significant differences were found between the score 
means of both groups in the pre-test, and of the comparison group in the pre- and post- tests. 
These results indicate that the use of the Introductory module might influence the experimental 
group test performance. 
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Students' performance in the final exam 

In the first trial, the mean score of the students in the General Chemistry final exam of 
the course was 1 16 and the standard deviation was 15.53. The General Chemistry professors 
constructed this exam, and the total score was 1 35. Significant correlation was found between 
the students' scores in pre-test and the final exam (r=0.75, p<0.01). Six out often students 
scored over the mean but all of them obtained total scores over 70% of the test. 

In the second semester, the total score of the final exam was 140. The experimental 
group mean score was 80.42 whereas mean of the comparison group was 67.78. No significant 
differences were found in the final exam mean scores of the experimental and comparison 
groups. However, the mean score of the experimental group was higher than of the comparison 
group. In the experimental group, 1 5 out of 32 students (47%) scored over the group mean. 

Students' performance in the General Chemistry Course 

In the first trial, the average percentage in the General Chemistry Course of the 10 
students who completed the module was approximately 77%. Nine out of ten students obtained 
an average grade over 70%. Only, the correlation between the pre-test scores and the course 
grade was significant and the highest (r=0.86, p <0.01). In the second trial, the course grade 
average of the experimental group was 70.37%. Seventeen students (53%) got an average 
percentage grade over the mean. 
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Conclusions 

The evaluation findings of the project Enhancement of problem solving and scientific 
reasoning skills through computerized General Chemistry modules reveal that the activities were 
conducted as planned during the first year of implementation: (1) order and purchase hardware, 
authoring language and other materials; (2) designing the Introductory module and translating to 
programming language; and (3) piloting (in two trials) the Introductory module. 

Overall, users’ opinion regarding the Introductory module features was highly positive. 
Between 70% and 100% of the students gave affirmative responses to the inquired features such 
as clear instructions and explanations, useful examples, easy vocabulary, helpful feedback, good 
screen format, sequence and fair amount of text. Most of the students (80%) evaluate the 
module as excellent and, particularly, they pointed out the relevance of the Problem Solving 
section in the General Chemistry course. They commented that this section was the most helpful 
because the problems were somewhat similar to those presented in the first part of the course. 
Although, the students praised the entire module and the instructional experience, they made 
several suggestions towards the module improvement. The suggestions were attended in the 
second trial, and programming bugs were solved in the revised version of the module. 

Despite of the above positive results of this evaluation, the differences in the pre- and 
post-test scores of the first trial group were not significant. Therefore, it is not possible to 
attribute the slight increment in the post-test scores to the students' work on the Introductory 
module. This result agrees with others related to the non-conclusive positive effect of CAI on 
student learning. However, it is important to point out the unexpected high and significant 
relation between the students pre-test scores, the final exam and the course grade. The sample of 
ten students who completed successfully the Introductory module obtained scores over 70% in 
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the final test and satisfactory course grades. 

The results of the second trial showed a better picture of the effect of the use of 
Introductory module towards improving student’s achievement in General Chemistry. The 
experimental group scored higher in the post- test and in the final exam than the comparison 
group. This result supports the use of the module for improve students’ performance in the 
General Chemistry course. Also, it agrees with others studies related to the positive relation of 
computer assisted instruction and student learning. However, it should be noticed that some 
extraneous variables might influence the scores of both groups such as (1) different teaching 
styles, (2) sample selection and assignment to the experimental and comparison group, 

(3) changes in testing, (4) students’ attitudes towards the treatment, (5) coverage of the course 
material in the semesters, (6) students taking simultaneously other related courses such as 
Calculus II and Physics, and (7) treatment briefly applied. Thus, the results should be taken 
carefully. 

On the basis of the evaluation results, some recommendations for improving the module 
future use and accessibility follows: 

1. To prepare a handout with directions for the users which may include how to enter 
into the module, to write the answers in scientific notation and to use the module's 
bottoms. 

2. To incorporate in the module a management system for monitoring its use, the 
frequency of student usage and how many times shoe he attempts to solve each 
problem. 

3. To gain faculty evaluation regarding the module features. 

4. To document systematically the activities of the project. 
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5 . To administered (through the module or in paper) one or two set of exercises to the 
students at the end of each section on the Introductory module. The student should 
receive quick feedback on his or her responses. This also serves to collect 
information about the student immediate performance after the use of the module. 

6. To train other General Chemistry faculty and teacher assistants in the use of the 
module. 

Overall, the results of this formative evaluation indicate that the pilot phase of the project 
was effectively implemented and according to the proposal. The students positively highlighted 
several features of the Introductory module and its use. Thus, in light of the above evidence the 
project was conducted in the right direction towards its outcomes. 
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